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This report details how charter schools are increasingly run by private, nonprofit 
management organizations called charter school management organizations (CMOs). The 
researchers find that most CMOs serve urban students from low- income families, operate 
small schools that offer more instructional time, and attract teachers loyal to each school’s 
mission, based on survey data and site visits. The authors conducted an impact analysis 
focused only on middle school grades, finding that a small fraction of CMO-run middle 
schools boosted achievement growth at notable levels. But on average, student 
performance in the CMO-run schools did not outpace achievement growth in other 
charters or in host districts for a statistically matched set of students. This review finds 
that the report offers an objective assessment of the comparative benefits for middle- 
school students of a highly select set of CMOs. It also helps to identify organizational 
features that operate in successful CMO-run schools that are modestly associated with 
stronger student growth in the middle grades. However, the authors downplay aspects of 
their methodology that resulted in significant selectivity concerning which CMOs were 
studied, raising guestions regarding the population of charter schools to which they hope 
to generalize. 
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Review of Charter-School Management 
Organizations: Diverse Strategies 
and Diverse Student Impacts 

Bruce Fuller , University of Calif ornia, Berkeley 



I. Introduction 

Charter school advocates— still promising that their institutions will outpace garden- 
variety public schools as the movement enters its third decade— struggle with a thorny 
dilemma. Major national studies have shown that the average charter- school student does 
not learn at a more rapid clip than peers who attend regular public schools. 1 Conseguently, 
rigorous charter enthusiasts are pushing to prune low-performing charter schools. This is 
difficult for any self- regulating guild— say, teacher unions, state legal bars, or the American 
Medical Association— when they try to ease out errant members in hopes of lifting average 
performance. The other strategy is to empirically identify the features of high performers 
that are paying off and spread adoption, in this case elevating student achievement. 

In this light, kudos are due the Gates and Walton foundations, along with one financing 
agent, the New School Venture Fund, for commissioning a long-term study of whether 
charters operated by charter management organizations (CMOs) display discrete, effective 
practices that lift student learning curves over time. If their effects are robust and can be 
exported to other charter schools, then the charter movement might finally realize its long- 
promised returns. 

The first report within this larger project is now out from Mathematica and the Center on 
Reinventing Public Education (CRPE), the first a reputable and usually hard-headed 
evaluation firm, in collaboration with the latter organization, a decidedly pro- market 
University of Washington think tank headed by Paul Hill. 2 Their report, with authorship 
credit given to a dozen researchers listed in alphabetical order (referred to in this review as 
Furgeson, et al.), is titled Charter-School Management Organizations: Diverse Strategies 
and Diverse Student Impacts. The report was initially released in November 2011, with an 
updated version released in J anuary 2012. 

It offers a rich description of schools serving the middle school grades that are run by 22 
different CMOs across the nation— that is, just 22 that could provide complete data on 
their middle- school students. Focused on this narrow slice of CMOs, the study describes 
organizational practices that sometimes differ from those of a matched sample of 
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traditional public schools. It also offers a rather tortured attempt to detect notable 
achievement advantages . 3 

Overall, this mostly objective look at charter- school firms is worth studying, especially as 
the Obama Administration pushes states to lift caps on the number of charter schools, and 
as researchers and policymakers ask whether these schools are actually yielding potent 
benefits for students. Further, this long-term study goes well beyond the celebrated CMOs 
that benefit from a great deal of private financing, such as Green Dot or Knowledge is 
Power (KIPP) charters, venturing further toward findings that could be generalized to the 
now far-flung, wildly mixed array of virtual and brick- and- mortar charter schools. The 
authors make a strident effort toward greater external validity, but end up pointing their 
microscope only at middle- school students who attend a guite select and small portion of 
all CMO-run charters. 



II. Findings and Conclusions of the Report 

Mathematica’s study yields two notable and empirically sound discoveries concerning 
CMOs: how these mostly nonprofit firms continue to grow in major cities and therefore 
mainly serve children of low- income families, and how they often deploy organizational 
practices that prior research suggests will lift students’ learning curves. Yet the report falls 
short when attempting to demonstrate achievement benefits for CMO-run schools, mostly 
yielding statistically insignificant differences with a matched comparison group of 
students, as well as null effects from CMO-run versus independent “mom and pop” charter 
schools. Given these findings, the report’s insistence on pressing results that would not 
likely survive rigorous peer review for a scholarly journal erodes its credibility. 

The report is organized around three intriguing guestions, each of which is discussed 
briefly below. The guestions, guoted from p. xxi of the report, are as follows : 4 

Characteristics and context. How guickly are CMOs growing? Which students and 
areas do they serve and what resources do they use? What are the practices and 
structures of CMOs? What state policies and other factors appear to influence the 
location and growth of CMOs? 

Impacts. What are the impacts of CMOs on student outcomes and to what extent do 
these impacts vary across CMOs? 

Promising practices. Which CMO practices and structures are positively associated 
with stronger growth curves for middle- school students? 

Who is Served by Charter Management Organizations? 

The rapid growth of CMO-managed charters is rightfully noted. “Attracting substantial 
philanthropic support, CMO schools have grown rapidly from encompassing about six 
percent of all charter schools in 2000 to about 17 percent of a much larger number of 
charter schools by 2009 (p. 1 ),” the authors state, drawing on Gary Miron’s annual study 
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tracking the growth and features of CMOs. 5 Chapter 2 then digs into the character of this 
growth, which has been powered by state policy and rising demand among parents in blue- 
collar and poor communities across the nation. 

Using Miron’s data, the authors report that some 130 CMOs were operating 816 semi- 
autonomous, publicly funded charter schools in 2009- 10, serving almost one-guarter 
million students, or about one- fifth of all children attending charter schools. 

The report’s authors then chose to restrict their study sample to CMOs running at least 
four schools in 2007 and meeting certain other criteria. 6 This narrowed the 130-plus CMOs 
down to 40 CMOs with 292 schools, concentrated mainly in Arizona, California, Ohio, 
Illinois, New York, Texas, and the District of Columbia. Of these 40, the authors identified 

The Mathematica team provides little information regarding the non- 
public funding of CMOs. 

26 CMOs that were running a middle school and then just 22 that agreed to provide 
sufficient data for the Mathematica analysis. J ust 1 in 6 of all schools run by this select set 
of CMOs was a middle school, totaling 66 separate schools. Readers, however, never learn 
how the eventual sample of 22 CMOs compares with the CMOs excluded from the study. 
This is emblematic of the authors’ seeming lack of concern over the steps they took in 
whittling down all CMOs to a presumably highly selective subset. 

CMO-run schools grew in number between 1999 and 2009, by about 20% each year, a 
more rapid increase than that of independent, non- CMO-run charter schools. But growth 
in both subsectors had slowed by the end of the decade. CMOs in the study’s narrower 
sample opened about one new school each year in their initial six years of operation, 
increasing to a rate of about two new schools per year in subseguent years. 

J ust under three- guarters of the CMOs that met the report’s inclusion criteria ran schools 
in cities, reflecting the progressive, pro-eguity aims of many founders and private 
benefactors. The schools run by the subset of 22 management firms primarily serve 
African American and Latino students: 91% from the two ethnic groups, compared with 
76% of all students in their host districts. Of CMO- middle- school students in the 
Mathematica sample, 7 in 10 gualified for subsidized meals, compared with 64% in host 
districts. 

CMO-run charters, however, tend to under serve special education students and those with 
limited English proficiency, the authors detail. So, now student- level selectivity also arises, 
nested within the severe whittling- down of CMOs. 

That said, the authors are later clear about how the baseline test scores of charter students 
are higher than the comparison group, additional evidence of student- level selectivity: 

Most CMOs attract somewhat higher achieving students of color relative to those 
served by their host districts. Thirteen of 22 CMOs in our sample serve black students 
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who had significantly higher average pre- entry reading test scores than the averages for 
their black peers in the host district... Likewise, the pre-entry reading scores of 
Hispanic students in 13 of 23 CMOs were significantly higher than Hispanic averages 
locally. The percentages are similar for reading test scores (p. 17). 7 

Overall, the report illuminates how CMOs, while generally serving urban students in 
lower- income communities, enroll students who enter with higher test scores than those of 
neighboring peers and accept fewer pupils with disabilities or limited English. 

Do These Charter Schools Display Innovative Organizational Features? 

One great service of this report is the next step taken by the authors: providing details on 
how the organizational features of these CMO-run charters differ from those of regular 
public schools in their host districts. In some cases, however, the reader cannot know 
whether these structural differences are driven by the CMO itself or by the principals and 
teachers these firms attract. That is, do CMO leaders set in place advantageous 
organizational practices, or is the trick to attract educators who focus on improvement? 
This is crucial when advancing lessons for urban school leaders. And the schools against 
which the sample schools are compared shift from all schools in host districts to, now, 
“nearby schools” (p. 21 ), which is confusing. Still, the organizational distinctions remain 
noteworthy. 

The CMO-run charters enroll just 389 students on average, compared with 982 in nearby 
regular public schools, a large difference not attributable to differing grade mixes. Average 
class sizes are smaller in CMO-run charters, about 21 pupils compared with just over 23 in 
neighborhood regular schools. Disaggregating by elementary, secondary schools would be 
helpful in future reports. 

The report found wide variation in per- pupil spending (for 39 of the 40 CMOs meeting 
inclusion criteria for this descriptive analysis), ranging from about $5,000 to more than 
$20,000 per pupil. The factors underlying this variability are never examined carefully. 

The authors do report a .61 correlation between per- pupil expenditures and per- pupil state 
funding, but this accounts for just one- third of the variation in spending across CMOs. So, 
ample private money appears to be in play. 

It’s disappointing that the Mathematica team provides little information regarding the 
non- public funding of CMOs. They do say, “In addition to state funding, some CMOs 
receive philanthropy and other funding. At least 9 of these CMOs spend more than $ 1,000 
per pupil beyond the amounts allocated from public sources and four CMOs spend more 
than $ 4,000 per pupil more” (p. 19 ). Given that at least one earlier study found that fully 
one- third of KIPP’s operating budget is covered by private donors , 8 this raises questions— 
never confronted in the report— about the replicability of an organizational model that 
relies on so much soft money. 

Mathematica’s Furgeson and colleagues also surveyed principals in schools run by 36 of 
the 40 CMOs meeting inclusion criteria (p. 86 ). “Of 292 CMO principals eligible for the 
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study, 76 percent responded to the survey. Among the 292 matched comparison principals, 
the response rate was 59 percent. Four CMOs eligible for the study declined to participate” 
(p. 25). So now we have yet another step toward narrowing selectivity in the study sample: 
the principal- reported data are derived from a non- random subsample of those working in 
schools run by a selected subset of the CMOs operating nationwide. 

Yet the principal- reported findings do usefully illuminate organizational features of this 
select group of charter schools, now compared with a “matched district school” (no longer 
a “nearby school”) (p. 87). The report notes that the unit of analysis is constrained to the 
36 CMOs participating in the principal survey, since schools are not independent units but 
are nested within CMOs. 

Charter principals reported more instructional time for their students— either longer 
school days or school years— when compared with their matched regular public schools: 
almost 1,400 hours of instruction on average, compared with just over 1,200 in traditional 
schools. The charter principals reported having more autonomy in selecting curricula 
packages and pedagogical strategies, with weaker central control by the CMO (this is in 
comparison to with principals’ peers in traditional public schools). This finding is to be 
expected, as it is aligned with the structure and purpose of chartering, but an interesting 
guestion remains as to whether, in this era of standards- based accountability, looser 
coupling of curricular decisions is truly preferable. 

Principals employed at CMO- run schools were more likely than their peers in regular 
schools to have specific student- behavior policies in place. (No data are reported on the 
incidence of “coaching” students with behavior problems to transfer out of charters— a 
concern that the researchers might explore in future work.) They also reported more 
freguent oversight by central -office managers, greater use of teacher mentoring and 
coaching, and more freguent use of student data in evaluating teachers. Perhaps most 
telling, charter principals were more likely to say that they select new teachers based on a 
sample classroom lesson and allegiance to the school’s mission, as opposed to the 
recruitment practices reported by regular- school principals, often constrained by district 
allocation of teachers based on seniority. 

Do CMO-run Charter Schools Yield Stronger Results? 

Furgeson and colleagues begin their pivotal chapter 4 with sound candor: “Overall average 
[student achievement] impacts appear to be positive, but they are not statistically 
significant” (p. 45). On average, they found no differences in achievement growth between 
students with matched propensities to enter a CMO-run charter relative to those with the 
propensity to not enter the “treatment”— that is, to enter a regular public school. 

This method of first modeling which students are more likely to select into a “treatment” 
or control condition— known as propensity score matching— does more thoroughly control 
for otherwise confounding and prior effects of family background. But propensity score 
matching yields more valid estimates of “treatment effects” (compared with ordinary least - 
sguares regression) only when rich data on possible confounders are available (e.g., home 
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or other factors that would influence achievement and the likelihood of entering a CMO- 
run charter). The authors had no such data at their disposal. So, propensity score 
matching— with such limited data— does not rule out omitted- variables bias. 

Nor did the authors find any discernible achievement- growth differences between students 
attending a CMO-run and those attending an independent charter school. 

If these results— the lack of mean differences among enrollees of CMO-run charters, 
independent charters, and a matched sample of regular public school pupils— were 
considered by a peer-reviewed journal, they would not likely see the light of day. Few 
journal editors are eager to publish null effects. But the authors push on— with some 
justification— to identify top- of- the- line CMO-run charters that do yield significant 
achievement- growth benefits, although only for middle- school students in their highly 
selective subset of CMO-run schools. The achievement analysis is conducted on a narrow 
set of 66 middle schools operated by just 22 CMOs that met the inclusion criteria, agreed 
to participate, and could provide longitudinal data for their students. These criteria likely 

The report’s unrelenting search for achievement effects in a small, 
selective subset of sampled CMOs erodes its credibility. 

create a selection bias, since firms meeting the latter two reguirements are likely to be 
doing a lot of other good things that those who refuse to participate or lack longitudinal 
data are not. These other good things remain unobserved and unmeasured by the research 
team but might well be driving positive results that the authors may improperly attribute 
to the CMO- operated status of the schools. 

Also, for students to be eligible for the achievement analysis, they must have entered the 
middle school in the first year of the grade cycle and have scores available for both a pre- 
entry year, to serve as a baseline, and at least one subseguent year. The number of CMO 
students who were enrolled for one, two, and three years (with reading scores), for 
example, is 18,769, 13,674, and 8,131, respectively. They are compared to all students in 
host districts. Again, there is a potential bias introduced by this otherwise- reasonable 
approach, since more transient students— the type excluded only from the charter side of 
the analysis— may well be lower scoring. A basic student attrition analysis would help to 
inform this threat to validity. 

Furgeson et al. do competently employ propensity- score matching to isolate school- 
specific effects. They note that just 16 of the 66 middle schools were over- subscribed, 
necessitating waiting lists and lottery selection of students, so guasi- experimental 
comparisons that take into account selection bias would introduce their own set of 
exclusion and bias problems. Propensity- score matching is a widely accepted alternative. 
But again, the assumption is that observed selection factors (say, the student’s race or 
language status) entered into the first- stage likelihood functions include most confounding 
factors that also predict student achievement. That is, as with any regression model, it is 
important to include, in order to control for, the factors that may be driving the outcome 
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independent of exposure to the “treatment condition” being studied— in the present case, 
attending a middle school run by a CMO. 

The authors do emphasize (p. 45) inclusion of the student’s prior test score as a predictor 
of selecting a charter school. But a variety of other, unobserved selection predictors could 
be at play, such as maternal education level, parental commitment to education, and time 
available to drive the child to a distant charter school— all factors that could help explain 
achievement advantages during the middle- school years. Without an adeguate set of 
confounders in the selection model, estimated average treatment effects with propensity- 
score matching fail to approximate a true experiment. Such limitations are not reasons to 
abandon the analysis; but the authors should inform their readers of these crucial 
constraints and alert readers that omitted variables may lead to over- estimation of alleged 
“treatment” effects, in this case attending a CMO- run charter school. 

The report states: 

In both reading and math, after two years of enrollment in CMOs, positive impacts are 
more common than negative impacts. Of the CMOs covered by the impact analysis, half 
( 11 of the 22) have positive impacts in math or reading while nine have negative 
impacts in one or both subjects (p. 45). 

This 11 versus 9 comparison, particularly in light of the non- significance of almost all 
mean differences, is not meaningful evidence of a charter- school advantage. Larry Hedges, 
among others, has long cautioned against using “vote counting” across studies or 
coefficients to make claims. 9 

The report estimates that 4 CMOs pushed reading scores up at least 0.20 of a standard 
deviation after two years of exposure, compared with matched students (p. 53), but 11 of 
the CMO- run schools showed flatter achievement change relative to matched students. 
Growth benefits in math scores are more impressive, with 7 CMOs yielding effect sizes over 
0.30 SD. This is something to write home about. Such results would accelerate children’s 
learning in middle- school by about one- fourth of a school year. But this upbeat news 
applies only to the rather select set of CMOs and their schools. Again, to what population 
of charter schools do the authors want us to generalize? 

Similarly, thej anuary 2012 updated report adds promising results on high school 
graduation rates. That is, students in the highly select subset of CMOs that survive the 
layers of inclusion criteria appear to graduate at higher rates than the matched comparison 
group. Again, the reader wonders about the generalizability of these findings. To what 
narrow slice of charter students do these findings apply? 

In short, highlighting the results of top performers in any distribution runs the risk of 
misleading readers about overall mean effects. What’s disappointing for charter -school 
adherents is the bottom line that the average effects of attending a CMO -run charter 
school are not significantly different from those of attending a regular public school. As 
Furgeson and colleagues conclude: 
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Although average two-year impacts are positive, they are not statistically significant at 
the five percent level. The average CMO’s two-year math impact is 0.11 and is 
marginally significant (p=0.08). Average impacts across CMOs for three- year science 
and social studies are positive and not statistically significant (p. 56). 

The authors also compared growth in middle- school student test scores for students in 
CMO-run and independent charter schools. They found “no patterns in the relative 
impacts of independent charter schools versus CMOs across the jurisdictions” (p. 55). 

In the final part of the study, the authors attempt to empirically relate organizational 
features of CMO-run charters to student achievement growth. Their a priori hypotheses 
are helpful in theorizing why charter schools might yield stronger achievement. They 
found that larger CMOs (operating more schools, serving more students than smaller 
CMOs) tend to yield significant achievement benefits at the middle- school level. The 
authors infer that “This might indicate that funders have had some success in supporting 
the expansion of CMOs that are more effective” (p. 58). It’s a sweet compliment for the 
foundations supporting the authors, but causality could flow in the opposite direction: 
more private money drives stronger learning opportunities, as capital is consolidated 
around a few boutigue firms. 

The report’s analysis also showed that, even though larger CMO networks displayed 
stronger results, rapid growth of CMO enrollments during the study period appears to 
suppress achievement gains. One possible explanation for this direction of causality 
offered by the report is that guality control arguably attenuates with greater size. 

Perhaps most importantly, Furgeson and his colleagues include some findings about 
instruction. They found that CMOs that provided more instructional time, emphasized 
mentoring and coaching of teachers, reviewed student data more intensively, and enforced 
a clear student- behavior policy showed significantly stronger achievement benefits. Again, 
dangers lurk about drawing strong causal inferences, but these are discrete organizational 
features that appear to explain the success of the narrow subset of CMOs that showed 
impressive results. They also potentially provide clear lessons for regular public middle 
schools. 

More work remains to be done in designing stronger causal research designs and in 
estimating the magnitude of any benefits from specific organizational features. Reported 
associations between school attributes and student impacts never exceed 0.20; that is, they 
explain less than 4 % of the variance in student growth (p. 65). Ideally, students’ growth 
curves would be more robust and variable, making it possible to determine which 
organizational attributes are driving any advantages. A less selective set of CMO-run 
schools would be desirable for this reason. As it is, we never learn whether charters are 
truly innovative— “thinking outside the box” organizationally— or if, instead, some CMOs 
are able to simply intensify structures and practices that we already know boost 
achievement, such as more instructional time and more efficient use of that time. 
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III. The Report’s Rationale for Its Findings and Conclusions 



The report combines elements of rigorous research with an implicit dash of advocacy. 
Regarding the former, Furgeson, et al. lay out a clear logic informing their investigation of 
whether CMOs yield greater achievement benefits than traditional public schools and 
independent, non- affiliated charters. They also usefully seek to identify organizational 
features that help explain achievement advantages in CMO-run charters. Their rationale 
for this important empirical exercise is clearly stated— and implemented with care and 
rigor overall. 

But at times the authors seem to channel the hopes and ideology of their funders. In one 
example of the advocacy elements of their report, the authors frame their study by saying: 

An extensive body of research suggests that variation in the performance of charter 
schools is wide but that high-performing charter schools can produce substantial 
positive achievement effects for their students. CMOs represent an attempt to produce 
the effects of high-performing charter schools on a larger scale (p. 45). 

This is a considerably rosier picture than that painted by earlier, well known studies that draw 
on two national data sets, which found that students attending charters fail to show test score 
growth at rates greater than their peers attending traditional public schools. 10 (With the 
exception of Mathematica’s own earlier study of middle schools, these studies are not even 
referenced in the new report.) By analyzing whether CMOs have been successful in the past, 
and under what conditions, the authors aim to assess whether further consolidation might 
yield benefits for students in the future. This holds implications for whether the returns on 20 
years of public investment in charter schools can ever actually pay off for students and 
taxpayers. 

Another pertinent guestion is whether a movement that began as a grassroots revolt 
against over- centralized, homogenous public schools really benefits from being directed 
and coordinated by a few foundation officials and their allies in Washington. This is not to 
be critical about this drift: the jury is still out. But as researchers dig deeper into the 
results associated with charter management firms, they should be more clear- eyed and 
candid about the disappointing results from charters overall and the political and 
institutional dynamics by which the “movement” is becoming rationalized and centrally 
controlled. 



IV. The Report’s Use of Research Literature 

The authors dodge the wider debate over whether charter schools, on average, outperform 
regular public schools after taking into account differences in student background. The 
Obama administration and major foundations are betting on the hope that charters may one 
day outperform regular public schools, despite the fact that, after two decades of the charter 
school movement, no generalizable evidence substantiates that hope. 
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Furgeson and colleagues do build from prior work examining whether particular types of 
charters, such as CMO-run organizations, or charters in particular places (e.g.. New York 
City) show discernible achievement advantages. The report’s literature review regarding 
methodological approaches is carefully crafted. In fact, this review is a must-read for any 
serious student of charter schools, in part for its consideration of alternative methods for 
assessing specific organizational ingredients. 

Yet as suggested above, there is room for improvement. Future reports from this research 
team should follow more standard practice to avoid excessively reducing their samples. At 
a minimum the research team should detail how the select subsamples compare with 
starting samples (and the universe) of CMOs and the schools they operate. 

In addition, the authors should give greater attention to the search for school features that 
yield achievement effects— features that might then be scaled up. From research on 
preschools to high school programs aimed at increasing college access, we know a great 
deal about the difficulties institutions have in spreading organizational changes that yield 
benefits for students. In the sphere of charter schools, this report’s contribution in 
pinpointing organizational features that appear to pay off for a handful of CMOs is 
noteworthy. But a key guestion is how to spread these organizational reforms to a wider 
set of schools— even to a broader range of charter schools— to increase their positive 
impact on student learning. 

V. Review of the Report’s Methods 

In many ways, the report’s methods are sound. For instance, the authors were quite careful 
when reporting the magnitude of achievement effects for differing sets of CMOS- 
important as policy makers and the reform community weigh the promise of charter 
schools relative to alternative reform strategies. 

The major methodological concern with this report has already been detailed above: the 
successive narrowing of the CMOs included in the study from the 130 -plus operating 
nationwide to 22 that met layers of inclusion criteria. Estimates of achievement effects are 
then further narrowed down to students in charter middle schools who began at the 
beginning of the grade cycle and for whom there were complete pre-test data. Once thus 
narrowed, the study presents impressive results for between 4 (reading outcome) and 7 
(math outcome) of the 22 CMOs. So, to whom or what can we generalize these results? 

The decision to estimate achievement effects only at the middle school is especially odd. The 
report justifies it by saying that no pre-test scores are available for elementary- age students, 
given that standardized testing doesn’t begin until grade 3 (although this is not the case in 
some states, like California, where testing begins in grade 2). Developmental psychologists 
have been using a variety of language and cognitive assessments for children as young as 3 
years old, which hold strong predictive validity . 11 

The report also claims that grade repetition is too great and subject-matter tests too 
numerous for an assessment of the effects of charter high schools— and analysts must focus 



http :/ / nepc. Colorado . edu/ thinktank/ review- charter- school- management 



10 of 14 



on the middle grades. This is a strange position to take, especially as charter advocates 
promise strong benefits from charter high schools. Further, the report’s own analysis focuses 
on students enrolled for two years in middle school, demonstrating the short period of time 
for which change scores can also be estimated (with prior achievement levels) for students 
attending elementary or high school. 

Another irregular decision on methodology was to report as statistically significant 
regression coefficients that are “significant” at the liberal level of p<.io, even though the 
conventional, more rigorous threshold of p<.05 was used in earlier chapters. The authors 
estimate a huge number of parameters, rendering the p<. 10 level as meaningless. This 
erodes the report’s credibility, signaling a hunt for some good news concerning 
achievement advantages for charter students. 

VI. Review of the Validity of the Report’s Findings and Conclusions 

The report’s most useful and generalizable findings can be found in its descriptive 
attributes of the nation’s schools run by charter management organizations. It is important 
to know that these CMOs— or at least those CMOs that met the report’s inclusion criteria— 
tend to serve children from low- income families and that a sizeable portion of these 
organizations display effective practices more consistently than neighboring regular public 
schools. These charter organizations have apparently learned from the past half-century of 
research on effective schools, and operationalized these lessons. They and their 
benefactors turn out to be quite committed to serving disadvantaged populations, 
something that has worried many of us since the movement’s inception in 1991 . 

Yet the report’s unrelenting search for achievement effects in a small, selective subset of 
sampled CMOs erodes its credibility. It does legitimately find a handful of CMO-run 
middle schools that show impressive achievement gains (over 0.30 SD). But on average 
CMO-run charters outperform neither regular public schools nor independent charter 
schools— as the authors themselves forthrightly acknowledge. Similarly, in the J anuary 
2012 update to their report, the authors drift into oversell, saying in their press release, 
“The study shows that some— but not all— CMOs substantially boost students’ chances of 
graduating from high school and enrolling in postsecondary education .” 12 Rather than 
“some” CMO-run schools, the authors should be more intellectually honest, reminding the 
reader that the original report yields findings that are generalizable to a small, highly 
selective slice of non- representative CMO-run schools. 

VII. Usefulness of the Report for Guidance of Policy and Practice 

The report’s findings offer useful information for the federal government and key 
foundations that have increasingly supported charter management organizations as a 
strategy for lifting the benefits of charter schools. We learn that a few CMOs run schools 
that outperform regular public schools serving statistically matched students. But on 
average CMO-run schools do not outperform their regular- school peers, and students 
attending independent charters did just as well. Essentially, each of these sectors or 
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segments of the schooling universe has a distribution of higher- and lower- performing 
schools, and on average they are about equal to one another. 

Support of charter schools— by political leaders and restless taxpayers— will likely wane if 
their comparative benefits remain difficult to discern. Twenty years after the first charters 
opened, the variability of their performance closely mirrors the unevenness of regular 
public schools. This report gets us a bit closer toward pinpointing highly effective CMOs 
and the practices or resources that may help to explain their efficacy. What is not known is 
whether these practices or resources can be exported with integrity to other charter 
companies or to the wider public school system. These questions should be squarely 
addressed in future research. 
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